UI Tools

Overview

The embed widget offers the ability to run tools with AI context or arguments, allowing you to display information or manipulate the DOM based on the customization provided to the tool. Additionally, it includes a set of built-in tools accessible via window.cmndBuiltInTools.

Built-in tools

The built-in tools are pre configured utilities that come with the embed widget. These tools are available by default unless you provide custom tools.

Below is a list of the built-in tools along with their descriptions.

show_form
Displays a form based on a provided JSON schema.
capture_signature
Captures a user's signature.
show_rating
Displays a rating input widget.
show_video_player
Displays a video player.
show_image_viewer
Displays an image viewer.

These tools can be utilized by providing a prompt related to the tool's functionality. For example, asking "Show me a login form" will render a login form.

Custom UI Tools

Custom UI tools allow you to extend the functionality of the embed widget by defining your own tools tailored to specific use cases. These tools can override the default built-in tools or work alongside them by leveraging window.cmndBuiltInTools. Custom tools provide flexibility and adaptability, enabling you to create unique experiences that align with your application's requirements.

warning

Providing custom UI tools overwrites the built-in tools, but you can extend your custom UI tools with the built-in tools using window.cmndBuiltInTools.

Defining a Custom UI Tool

Custom UI tools allow you to extend the functionality of the embed widget to suit your specific requirements. Each custom tool must have a functionType set to ui and follow a consistent structure. Below is a guide on how to define a custom UI tool, explained using a sample tool.

Example: Custom UI Tool Definition

Here’s a complete example of a custom tool: Example functionality:

Updates the background color of the page.
Updates the innerHTML of the referenced element with a confirmation message.

// script.js
{
  name: "change_background_color", // Unique tool name
  description: "Changes the background color of the page", // Tool description
  category: "UI", // Main category
  subcategory: "BG", // Subcategory
  functionType: "ui", // Function type (always "ui")
  dangerous: false, // Safe tool indicator
  associatedCommands: [], // Additional commands (if any)
  prerequisites: [], // Prerequisites (if any)
  argumentSchema: { // Arguments required by the tool
    type: "object",
    properties: {
      bgColor: {
        type: "string", // The background color argument
        minLength: 1, // Minimum length validation
        errorMessage: {
          required: "Background Color is required", // Error message if missing
        },
      },
    },
    required: ["bgColor"], // Required arguments
  },
  rerun: true, // Allows rerunning the tool
  rerunWithDifferentParameters: true, // Allows rerunning with different inputs
  capturesUserInput: true, // Captures user input
  runCmd: ({ functionArgs }, ref) => {
    const { bgColor } = functionArgs;
    const messageContainer = ref;

    if (messageContainer) {
      messageContainer.innerHTML = `Background color changed to ${bgColor}`;
    }

    document.body.style.backgroundColor = bgColor; // Change background color
  },
}

Key Properties of a Custom UI Tool

name
A unique identifier for the tool.
Example: "change_background_color"
description
A short description of what the tool does.
Example: "Changes the background color of the page"
dangerous
Indicates whether the tool performs potentially harmful actions.
Example: false
associatedCommands and prerequisites
Define additional commands or prerequisites required by the tool. These can be left empty if not needed.
Example: []
argumentSchema
Defines the arguments required to run the tool in JSON schema format:
- properties: Lists the arguments.
  Example: bgColor (a required string).
- required: Specifies mandatory arguments.
- errorMessage: Custom error messages for validation.
rerun and rerunWithDifferentParameters
Specifies if the tool can be rerun with the same or modified parameters.
Example: Both set to true.
capturesUserInput
This parameter denotes whether the given UI tool is responsible for capturing input from the user. For example, if your UI tool is showing an input field that is filled by the user, then you should set this parameter to true. If your UI tool is only showing information but is not responsible for capturing input e.g showing an image, or a chart you can set this to false. When this field is set to true, chatbot will display the UI tool and respond back to the user acknowledging the successful run of the tool. If it's set to false, it will only dislay the tool but it won't respond. Example: true
runCmd
The function that implements the tool’s functionality. It receives:
- functionArgs: Contains the tool's arguments (e.g., bgColor).
- captureResults: Callback function. You must call this function with the input you captured from the user as a string. If your tool doesn't capture user input, you can call it with an informational message e.g "Displayed plot chart to the user."
- previousRunResults: If the tool was run previously, it contains the string captureResults was called with. You can check this to prevent re-evaluating the same tool twice.
- ref: A reference to the DOM element where you can render your own UI elements within the chat box message card.

Usage

We do not need to register built-in tools, but we need to register custom UI tools using the registerTools command

// script.js
window.cmndChat(
  "registerTools",
  {
    ui_tools: [
      {
         name: "change_background_color",
         // ... other values
      },
      {
         name: "navigate_to_page",
         //... other values
      }
    ]
  }
);

To extend custom UI tools with built-in tools you will need to include the built-in tools using window.cmndBuiltInTools. Below is a example

// script.js
window.cmndChat(
  "registerTools",
   {
    ui_tools: [
      window.cmndBuiltInTools.capture_signature,
      window.cmndBuiltInTools.show_form,
      {
         name: "change_background_color",
         // ... other values
      },
      {
         name: "navigate_to_page",
         //... other values
      }
    ]
   }
);

Types of Custom UI Tools

There are four variations of UI tools that you can implement and use:

Rendered in the chat box as a message card, without capturing user input
These tools are displayed in the chat box as message cards and do not require user input. They must use the ref parameter within their runCmd method. Example: plot_graph
Rendered in the chat box as a message card, capturing user input
These tools are displayed in the chat box as message cards and require user input. They must use the ref and captureResults parameters within their runCmd method.
Example: capture_user_name
Not rendered in the chat box, without capturing user input
These tools are not displayed in the chat box and do not require user input. They might access the DOM directly if necessary. These tools only use the functionArgs parameter within their runCmd method.
Example: change_background_color
Not rendered in the chat box, capturing user input
These tools are not displayed in the chat box but do require user input. They might access the DOM directly if necessary. These tools use the functionArgs and captureResults parameters within their runCmd method.
Example: show_prompt_window

Example: Rendered in the chat box as a message card, without capturing user input

{
  name: "plot_graph",
  description: "Plots a graph on the page using Chart.js",
  dangerous: false,
  associatedCommands: [],
  prerequisites: [],
  argumentSchema: {
    type: "object",
    properties: {
      type: {
        type: "string",
        enum: ["line", "bar", "pie", "doughnut"],
        errorMessage: {
          required: "Graph type is required",
          enum:
            "Graph type must be one of: line, bar, pie, or doughnut",
        },
      },
      data: {
        type: "object",
        properties: {
          labels: {
            type: "array",
            items: { type: "string" },
            errorMessage: {
              required: "Labels for the graph are required",
            },
          },
          datasets: {
            type: "array",
            items: {
              type: "object",
              properties: {
                label: { type: "string" },
                data: {
                  type: "array",
                  items: { type: "number" },
                  errorMessage: {
                    required: "Dataset values are required",
                  },
                },
                backgroundColor: {
                  type: "array",
                  items: { type: "string" },
                },
                borderColor: {
                  type: "array",
                  items: { type: "string" },
                },
                borderWidth: { type: "number" },
              },
            },
          },
        },
        required: ["labels", "datasets"],
      },
    },
    required: ["type", "data"],
  },
  rerun: true,
  rerunWithDifferentParameters: true,
  capturesUserInput: true,
  runCmd: (
    { functionArgs, previousRunResults, captureResults },
    ref
  ) => {
    const { type, data } = functionArgs;

    // Check if Chart.js is already loaded
    if (!window.Chart) {
      console.log("Chart.js not found. Loading dynamically...");

      // Create a script element to load Chart.js
      const script = document.createElement("script");
      script.src = "https://cdn.jsdelivr.net/npm/chart.js";
      script.onload = () => {
        console.log("Chart.js loaded successfully.");
        renderChart();
      };
      script.onerror = () => {
        console.error("Failed to load Chart.js.");
      };

      document.head.appendChild(script);
    } else {
      renderChart();
    }

    function renderChart() {
      // Ensure the ref element exists
      if (!ref) {
        console.error(
          "No reference element provided for the chart."
        );
        return;
      }

      // Check if the ref element already contains a canvas, if so, reuse it
      let canvas = ref.querySelector("canvas");
      if (!canvas) {
        // Create a new canvas element
        canvas = document.createElement("canvas");
        ref.appendChild(canvas);
      }

      // Destroy any existing chart on the canvas
      if (ref.chartInstance) {
        ref.chartInstance.destroy();
      }

      // Render the chart using Chart.js
      const ctx = canvas.getContext("2d");
      ref.chartInstance = new Chart(ctx, {
        type,
        data,
        options: {
          responsive: true,
          maintainAspectRatio: false,
          plugins: {
            legend: {
              position: "top",
            },
          },
        },
      });

      if (!previousRunResults) {
        captureResults(
          "Graph successfully plotted inside the provided ref."
        );
      }
    }
  },
}

Example: Rendered in the chat box as a message card, capturing user input

{
  name: "capture_user_name",
  description: "Renders a text field to capture the user's name",
  dangerous: false,
  associatedCommands: [],
  prerequisites: [],
  argumentSchema: {
    type: "object",
    properties: {},
  },
  rerun: true,
  rerunWithDifferentParameters: true,
  capturesUserInput: true,
  runCmd: ({ previousRunResults, captureResults }, ref) => {
    // Ensure the ref element exists
    if (!ref) {
      console.error("No reference element provided.");
      return;
    }

    // Clear the content of the ref
    ref.innerHTML = "";

    if (previousRunResults) {
      // Display the user's name if previous results exist
      const greeting = document.createElement("span");
      greeting.textContent = `Hello, ${previousRunResults}`;
      greeting.style.fontSize = "1.2rem";
      greeting.style.fontWeight = "bold";
      ref.appendChild(greeting);
    } else {
      // Create a container for the input field and button
      const container = document.createElement("div");
      container.style.margin = "1rem";

      // Create a label and input field
      const input = document.createElement("input");
      input.type = "text";
      input.placeholder = "Enter your name";
      input.style.marginRight = "0.5rem";

      // Create a submit button
      const button = document.createElement("button");
      button.textContent = "Submit";
      button.style.padding = "0.5rem 1rem";

      // Append elements to the container
      container.appendChild(input);
      container.appendChild(button);
      ref.appendChild(container);

      // Add event listener for the submit button
      button.addEventListener("click", () => {
        const name = input.value.trim();
        if (name) {
          captureResults(name);

          // Replace the input and button with the greeting
          ref.innerHTML = "";
          const greeting = document.createElement("span");
          greeting.textContent = `Hello, ${name}`;
          greeting.style.fontSize = "1.2rem";
          greeting.style.fontWeight = "bold";
          ref.appendChild(greeting);
        } else {
          alert("Please enter a name.");
        }
      });
    }
  }
}

Example: Not rendered in the chat box, without capturing user input

{
  name: "change_background_color",
  description: "Changes the background color of the page",
  category: "UI",
  subcategory: "BG",
  functionType: "ui",
  dangerous: false,
  associatedCommands: [],
  prerequisites: [],
  argumentSchema: {
    type: "object",
    properties: {
      bgColor: {
        type: "string",
        minLength: 1,
        errorMessage: {
          required: "Background Color is required",
        },
      },
    },
    required: ["bgColor"],
  },
  rerun: true,
  rerunWithDifferentParameters: true,
  capturesUserInput: false,
  runCmd: (
    { functionArgs, previousRunResults, captureResults },
    ref
  ) => {
    const { bgColor } = functionArgs;
    if (!previousRunResults) {
      captureResults(`Background color changed to ${bgColor}`);
    }
    const messageContainer = ref;
    if (messageContainer) {
      console.log("we got message container", messageContainer);
      messageContainer.innerHTML = `Background color changed to ${bgColor}`;
    }
    document.body.style.backgroundColor = bgColor;
  }
}

Example: Not rendered in the chat box, capturing user input

{
  name: "show_prompt_window",
  description:
    "Prompts the user with a message and captures their input",
  category: "UI",
  subcategory: "Prompt",
  functionType: "ui",
  dangerous: false,
  associatedCommands: [],
  prerequisites: [],
  argumentSchema: {
    type: "object",
    properties: {
      message: {
        type: "string",
        minLength: 1,
        errorMessage: {
          required: "Prompt message is required",
        },
      },
    },
    required: ["message"],
  },
  rerun: true,
  rerunWithDifferentParameters: true,
  capturesUserInput: true,
  runCmd: (
    { functionArgs, previousRunResults, captureResults },
    ref
  ) => {
    if (previousRunResults) {
      return;
    }
    const { message } = functionArgs;

    // Prompt the user with the provided message
    const userInput = prompt(message);

    // If user input is captured, store the result
    if (userInput !== null) {
      captureResults(`User entered: ${userInput}`);
    } else {
      captureResults("User cancelled the prompt.");
    }

    // If a reference element is provided, display the user input there
    const messageContainer = ref;
    if (messageContainer) {
      messageContainer.innerHTML =
        userInput !== null
          ? `User input: ${userInput}`
          : "Prompt was cancelled by the user.";
    }

    // Return the user input for further processing
    return userInput;
  }
}

Overview​

Built-in tools​

Custom UI Tools​

Defining a Custom UI Tool​

Example: Custom UI Tool Definition​

Key Properties of a Custom UI Tool​

Usage​

Types of Custom UI Tools​

Example: Rendered in the chat box as a message card, without capturing user input​

Example: Rendered in the chat box as a message card, capturing user input​

Example: Not rendered in the chat box, without capturing user input​

Example: Not rendered in the chat box, capturing user input​